Surgial optical zoom system

ABSTRACT

Methods and systems for controlling a surgical microscope. Moveable optics of the surgical microscope are controlled using two sets of control parameters, to reduce jitter and image instability. Shifts in the image due to changes in temperature or due to the use of optical filter can also be compensated. Misalignment between the mechanical axis and the optical axis of the surgical microscope can also be corrected.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority under 35U.S.C. § 120 from U.S. patent application Ser. No. 16/685,554 filed onNov. 15, 2019, which itself is a continuation U.S. patent applicationSer. No. 16/152,264 filed on Oct. 4, 2018, which claims priority fromCanadian Patent Application No. 2,981,726, filed Oct. 6, 2017, entitled“SURGICAL OPTICAL ZOOM SYSTEM”, the entirety of which is herebyincorporated by reference.

FIELD

The present disclosure is generally related to optical imaging systems,including optical imaging systems suitable for use in image guidedmedical procedures, and for medical procedures requiring high opticalzoom.

BACKGROUND

Surgical microscopes are often used during surgical procedures toprovide a detailed or magnified view of the surgical site. Often, a highoptical zoom is required while maintaining a large stand-off distance.Conventional surgical microscopes may exhibit noticeable image jitter athigh zoom, which is undesirable. Further, insufficient holding force forthe optical carriage may result in an unstable image.

As well, minor misalignment between the mechanical axis and optical axisof the imaging system, which may otherwise be acceptable or negligibleat lower zoom levels, may cause positioning errors at high zoom levels.

SUMMARY

In some aspects, there is provided a surgical microscope for capturingan image of a target during a surgical procedure. The surgicalmicroscope includes an optical assembly including at least one moveableoptics, and an actuator for positioning the moveable optics. Theactuator includes a pulley system for moving the optics along a set ofrails. The surgical microscope also includes a sensor for detecting theposition of the moveable optics, a controller for controlling theactuator in response to received control input, and a camera forcapturing the image of the target from the optical assembly. Thecontroller is configured to receive control input indicating a targetposition for the moveable optics. The controller is also configured tocontrol the actuator to position the moveable optics towards the targetposition. The actuator is controlled according to a first set of controlparameters. The controller is also configured to, upon receiving signalsfrom the sensor indicating that the moveable optics is within athreshold range of the target position, switch to a second set ofcontrol parameters for controlling the actuator. The controller is alsoconfigured to control the actuator to maintain the moveable optics atthe target position at steady state.

In some examples, the surgical microscope may include a filter wheelthat is positionable by a filter wheel actuator to position a selectableoptical filter in an optical path of the optics.

In some examples, use of the selectable optical filter may cause a shiftin the optical path. The controller may be further configured todetermine a compensation amount to adjust the target position for themoveable optics, to compensate for the shift, and to adjust the targetposition accordingly.

In some examples, the surgical microscope may include a temperaturesensor. The controller may be further configured to determine acompensation amount to adjust the target position for the moveableoptics, to compensate for temperature-dependent shift in an optical pathof the optics, and to adjust the target position accordingly.

In some aspects, there is provided a method for controlling a surgicalmicroscope. The method includes receiving control input indicating atarget position for a moveable optics of the surgical microscope. Themethod also includes controlling an actuator of the moveable optics toposition the moveable optics towards the target position. The actuatoris controlled according to a first set of control parameters. Theactuator includes a pulley system for moving the optics along a set ofrails. The method also includes, upon receiving signals from a positionsensor indicating that the moveable optics is within a threshold rangeof the target position, switching to a second set of control parametersfor controlling the actuator. The method also includes controlling theactuator to maintain the moveable optics at the target position atsteady state.

In some examples, when an optical filter is positioned in an opticalpath of the optics, the method may include determining a compensationamount to adjust the target position for the moveable optics, tocompensate for a shift in the optical path caused by the optical filter,and adjusting the target position accordingly.

In some examples, the method may include receiving information from atemperature sensor indicating a surrounding temperature of the optics.The method may further include determining a compensation amount toadjust the target position for the moveable optics, to compensate for atemperature-dependent shift in an optical path of the optics, andadjusting the target position accordingly.

In some aspects, there is provided a method of correcting formisalignment between a mechanical axis of a surgical microscope and anoptical axis of the surgical microscope. The mechanical axis is definedby a housing of the surgical microscope and the optical axis is definedby an optical assembly of the surgical microscope. The method includesreceiving control input to move the surgical microscope to a targetworking distance from a target. The method also includes applying acorrection matrix to transform the target working distance from anoptical axis frame of reference to a mechanical axis frame of reference.The correction matrix contains correction terms to correct formisalignment between the optical axis and the mechanical axis. Themethod also includes moving the surgical microscope according to thetransformed working distance.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made, by way of example, to the accompanyingdrawings which show example embodiments of the present application, andin which:

FIG. 1 shows an example navigation system to support image guidedsurgery;

FIG. 2 is a diagram illustrating system components of an examplenavigation system;

FIG. 3 is a block diagram illustrating an example control and processingsystem that may be used in the example navigation systems of FIGS. 1 and2;

FIG. 4A is a flow chart illustrating an example method involved in asurgical procedure that may be implemented using the example navigationsystems of FIGS. 1 and 2;

FIG. 4B is a flow chart illustrating an example method of registering apatient for a surgical procedure as outlined in FIG. 4A;

FIG. 5 shows the use of an example optical imaging system during amedical procedure;

FIGS. 6 and 7 are different perspective views of an example opticalimaging system;

FIG. 8 is a block diagram of an example optical imaging system;

FIG. 9 is a block diagram illustrating an example control loop for anexample optical imaging system;

FIG. 10 is a flowchart illustrating an example method of controlling anoptical imaging system;

FIG. 11 is a flowchart illustrating an example method of setting controlparameters for an optical imaging system;

FIG. 12 is a diagram illustrating an exaggerated misalignment of theoptical axis in an optical imaging system;

FIG. 13 is a perspective view of an example calibration apparatus forcalibrating the optical axis of an optical imaging system;

FIG. 14 is a flowchart illustrating an example method of calibration foran optical imaging system; and

FIG. 15 is a diagram illustrating the shift in optical path caused by anoptical filter;

FIG. 16 is a flowchart illustrating an example method of compensatingfor temperature-related or optical filter-related shifts in an opticalimaging system.

Similar reference numerals may have been used in different figures todenote similar components.

DESCRIPTION OF EXAMPLE EMBODIMENTS

The systems and methods described herein may be useful in the field ofneurosurgery, including oncological care, neurodegenerative disease,stroke, brain trauma and orthopedic surgery. The teachings of thepresent disclosure may be applicable to other conditions or fields ofmedicine. It should be noted that while the present disclosure describesexamples in the context of neurosurgery, the present disclosure may beapplicable to other surgical procedures that may use intraoperativeoptical imaging.

Various example apparatuses or processes will be described below. Noexample embodiment described below limits any claimed embodiment and anyclaimed embodiments may cover processes or apparatuses that differ fromthose examples described below. The claimed embodiments are not limitedto apparatuses or processes having all of the features of any oneapparatus or process described below or to features common to multipleor all of the apparatuses or processes described below. It is possiblethat an apparatus or process described below is not part of any claimedembodiment.

Furthermore, numerous specific details are set forth in order to providea thorough understanding of the disclosure. However, it will beunderstood by those of ordinary skill in the art that the embodimentsdescribed herein may be practiced without these specific details. Inother instances, well-known methods, procedures and components have notbeen described in detail so as not to obscure the embodiments describedherein.

As used herein, the terms, “comprises” and “comprising” are to beconstrued as being inclusive and open ended, and not exclusive.Specifically, when used in the specification and claims, the terms,“comprises” and “comprising” and variations thereof mean the specifiedfeatures, steps or components are included. These terms are not to beinterpreted to exclude the presence of other features, steps orcomponents.

As used herein, the term “exemplary” or “example” means “serving as anexample, instance, or illustration,” and should not be construed aspreferred or advantageous over other configurations disclosed herein.

As used herein, the terms “about”, “approximately”, and “substantially”are meant to cover variations that may exist in the upper and lowerlimits of the ranges of values, such as variations in properties,parameters, and dimensions. In one non-limiting example, the terms“about”, “approximately”, and “substantially” may be understood to meanplus or minus 10 percent or less.

Unless defined otherwise, all technical and scientific terms used hereinare intended to have the same meaning as commonly understood by one ofordinary skill in the art. Unless otherwise indicated, such as throughcontext, as used herein, the following terms are intended to have thefollowing meanings:

As used herein, the phrase “access port” refers to a cannula, conduit,sheath, port, tube, or other structure that is insertable into asubject, in order to provide access to internal tissue, organs, or otherbiological substances. In some embodiments, an access port may directlyexpose internal tissue, for example, via an opening or aperture at adistal end thereof, and/or via an opening or aperture at an intermediatelocation along a length thereof. In other embodiments, an access portmay provide indirect access, via one or more surfaces that aretransparent, or partially transparent, to one or more forms of energy orradiation, such as, but not limited to, electromagnetic waves andacoustic waves.

As used herein the phrase “intraoperative” refers to an action, process,method, event or step that occurs or is carried out during at least aportion of a medical procedure. Intraoperative, as defined herein, isnot limited to surgical procedures, and may refer to other types ofmedical procedures, such as diagnostic and therapeutic procedures.

Some embodiments of the present disclosure relate to minimally invasivemedical procedures that are performed via an access port, wherebysurgery, diagnostic imaging, therapy, or other medical procedures (e.g.minimally invasive medical procedures) are performed based on access tointernal tissue through the access port. The present disclosure appliesequally well to other medical procedures performed on other parts of thebody, as well as to medical procedures that do not use an access port.Various examples of the present disclosure may be generally suitable foruse in any medical procedure that may use optical imaging systems, forexample any medical procedure that may benefit from havingintraoperative imaging at high zoom level.

In the example of a port-based surgery, a surgeon or robotic surgicalsystem may perform a surgical procedure involving tumor resection inwhich the residual tumor remaining after is minimized, while alsominimizing the trauma to the intact white and grey matter of the brain.In such procedures, trauma may occur, for example, due to contact withthe access port, stress to the brain matter, unintentional impact withsurgical devices, and/or accidental resection of healthy tissue. A keyto minimizing trauma is ensuring that the surgeon performing theprocedure has the best possible view of the surgical site of interest ata sufficiently high zoom level, with a stable and clear image. At thesame time, the imaging system should maintain a sufficient stand-offdistance from the site of interest, to avoid contamination and also toavoid obstructing the surgeon.

In FIG. 1, an exemplary navigation system environment 200 is shown,which may be used to support navigated image-guided surgery. As shown inFIG. 1, surgeon 201 conducts a surgery on a patient 202 in an operatingroom (OR) environment. A medical navigation system 205 may include anequipment tower, tracking system, displays and tracked instruments toassist the surgeon 201 during his procedure. An operator 203 may also bepresent to operate, control and provide assistance for the medicalnavigation system 205.

FIG. 2 shows a diagram illustrating an example medical navigation system205 in greater detail. The disclosed optical imaging system may be usedin the context of the medical navigation system 205. The medicalnavigation system 205 may include one or more displays 206, 211 fordisplaying a video image, an equipment tower 207, and a positioningsystem 208, such as a mechanical arm, which may support an opticalimaging system 500 (which may include an optical scope). One or more ofthe displays a, 211 may include a touch-sensitive display for receivingtouch input. The equipment tower 207 may be mounted on a frame (e.g., arack or cart) and may contain a power supply and a computer orcontroller that may execute planning software, navigation softwareand/or other software to manage the positioning system 208 and/or one ormore instruments tracked by the navigation system 205. In some examples,the equipment tower 207 may be a single tower configuration operatingwith dual displays 206, 211, however other configurations may also exist(e.g., dual tower, single display, etc.). Furthermore, the equipmenttower 207 may also be configured with a universal power supply (UPS) toprovide for emergency power, in addition to a regular AC adapter powersupply.

A portion of the patient's anatomy may be held in place by a holder. Forexample, as shown the patient's head and brain may be held in place by ahead holder 217. An access port 12 and associated introducer 210 may beinserted into the head, to provide access to a surgical site in thehead. The imaging system 500 may be used to view down the access port 12at a sufficient magnification to allow for enhanced visibility down theaccess port 12. The output of the imaging system 500 may be received byone or more computers or controllers to generate a view that may bedepicted on a visual display (e.g., one or more displays 206, 211).

In some examples, the navigation system 205 may include a trackedpointer 222. The tracked pointer 222, which may include markers 212 toenable tracking by a tracking camera 213, may be used to identify points(e.g., fiducial points) on a patient. An operator, typically a nurse orthe surgeon 201, may use the tracked pointer 222 to identify thelocation of points on the patient 202, in order to register the locationof selected points on the patient 202 in the navigation system 205. Itshould be noted that a guided robotic system with closed loop controlmay be used as a proxy for human interaction. Guidance to the roboticsystem may be provided by any combination of input sources such as imageanalysis, tracking of objects in the operating room using markers placedon various objects of interest, or any other suitable robotic systemguidance techniques.

Tracking markers 212 may be connected to the introducer 210 for trackingby the tracking camera 213, which may provide positional information ofthe introducer 210 from the navigation system 205. In some examples, thetracking markers 212 may be alternatively or additionally attached tothe access port 12. In some examples, the tracking camera 213 may be a3D infrared optical tracking stereo camera similar to one made byNorthern Digital Imaging (NDI). In some examples, the tracking camera213 may be instead an electromagnetic system (not shown), such as afield transmitter that may use one or more receiver coils located on thetool(s) to be tracked. A known profile of the electromagnetic field andknown position of receiver coil(s) relative to each other may be used toinfer the location of the tracked tool(s) using the induced signals andtheir phases in each of the receiver coils. Operation and examples ofthis technology is further explained in Chapter 2 of “Image-GuidedInterventions Technology and Application,” Peters, T.; Cleary, K., 2008,ISBN: 978-0-387-72856-7, incorporated herein by reference. Location dataof the positioning system 208 and/or access port 12 may be determined bythe tracking camera 213 by detection of the tracking markers 212 placedon or otherwise in fixed relation (e.g., in rigid connection) to any ofthe positioning system 208, the access port 12, the introducer 210, thetracked pointer 222 and/or other tracked instruments. The trackingmarker(s) 212 may be active or passive markers. A display 206, 211 mayprovide an output of the computed data of the navigation system 205. Insome examples, the output provided by the display 206, 211 may includeaxial, sagittal and coronal views of patient anatomy as part of amulti-view output.

The active or passive tracking markers 212 may be placed on tools (e.g.,the access port 12 and/or the imaging system 500) to be tracked, todetermine the location and orientation of these tools using the trackingcamera 213 and navigation system 205. The markers 212 may be captured bya stereo camera of the tracking system to give identifiable points fortracking the tools. A tracked tool may be defined by a grouping ofmarkers 212, which may define a rigid body to the tracking system. Thismay in turn be used to determine the position and/or orientation in 3Dof a tracked tool in a virtual space. The position and orientation ofthe tracked tool in 3D may be tracked in six degrees of freedom (e.g.,x, y, z coordinates and pitch, yaw, roll rotations), in five degrees offreedom (e.g., x, y, z, coordinate and two degrees of free rotation),but preferably tracked in at least three degrees of freedom (e.g.,tracking the position of the tip of a tool in at least x, y, zcoordinates). In typical use with navigation systems, at least threemarkers 212 are provided on a tracked tool to define the tool in virtualspace, however it is known to be advantageous for four or more markers212 to be used.

Camera images capturing the markers 212 may be logged and tracked, by,for example, a closed circuit television (CCTV) camera. The markers 212may be selected to enable or assist in segmentation in the capturedimages. For example, infrared (IR)-reflecting markers and an IR lightsource from the direction of the camera may be used. An example of suchan apparatus may be tracking devices such as the Polaris® systemavailable from Northern Digital Inc. In some examples, the spatialposition and orientation of the tracked tool and/or the actual anddesired position and orientation of the positioning system 208 may bedetermined by optical detection using a camera. The optical detectionmay be done using an optical camera, rendering the markers 212 opticallyvisible.

In some examples, the markers 212 (e.g., reflectospheres) may be used incombination with a suitable tracking system, to determine the spatialpositioning position of the tracked tools within the operating theatre.Different tools and/or targets may be provided with respect to sets ofmarkers 212 in different configurations. Differentiation of thedifferent tools and/or targets and their corresponding virtual volumesmay be possible based on the specification configuration and/ororientation of the different sets of markers 212 relative to oneanother, enabling each such tool and/or target to have a distinctindividual identity within the navigation system 205. The individualidentifiers may provide information to the system, such as informationrelating to the size and/or shape of the tool within the system. Theidentifier may also provide additional information such as the tool'scentral point or the tool's central axis, among other information. Thevirtual tool may also be determinable from a database of tools stored inor provided to the navigation system 205. The markers 212 may be trackedrelative to a reference point or reference object in the operating room,such as the patient 202.

Various types of markers may be used. The markers 212 may all be thesame type or may include a combination of two or more different types.Possible types of markers that could be used may include reflectivemarkers, radiofrequency (RF) markers, electromagnetic (EM) markers,pulsed or un-pulsed light-emitting diode (LED) markers, glass markers,reflective adhesives, or reflective unique structures or patterns, amongothers. RF and EM markers may have specific signatures for the specifictools they may be attached to. Reflective adhesives, structures andpatterns, glass markers, and LED markers may be detectable using opticaldetectors, while RF and EM markers may be detectable using antennas.Different marker types may be selected to suit different operatingconditions. For example, using EM and RF markers may enable tracking oftools without requiring a line-of-sight from a tracking camera to themarkers 212, and using an optical tracking system may avoid additionalnoise from electrical emission and detection systems.

In some examples, the markers 212 may include printed or 3D designs thatmay be used for detection by an auxiliary camera, such as a wide-fieldcamera (not shown) and/or the imaging system 500. Printed markers mayalso be used as a calibration pattern, for example to provide distanceinformation (e.g., 3D distance information) to an optical detector.Printed identification markers may include designs such as concentriccircles with different ring spacing and/or different types of bar codes,among other designs. In some examples, in addition to or in place ofusing markers 212, the contours of known objects (e.g., the side of theaccess port 12) could be captured by and identified using opticalimaging devices and the tracking system.

A guide clamp 218 (or more generally a guide) for holding the accessport 12 may be provided. The guide clamp 218 may allow the access port12 to be held at a fixed position and orientation while freeing up thesurgeon's hands. An articulated arm 219 may be provided to hold theguide clamp 218. The articulated arm 219 may have up to six degrees offreedom to position the guide clamp 218. The articulated arm 219 may belockable to fix its position and orientation, once a desired position isachieved. The articulated arm 219 may be attached or attachable to apoint based on the patient head holder 217, or another suitable point(e.g., on another patient support, such as on the surgical bed), toensure that when locked in place, the guide clamp 218 does not moverelative to the patient's head.

In a surgical operating room (or theatre), setup of a navigation systemmay be relatively complicated; there may be many pieces of equipmentassociated with the surgical procedure, as well as elements of thenavigation system 205. Further, setup time typically increases as moreequipment is added. To assist in addressing this, the navigation system205 may include two additional wide-field cameras to enable videooverlay information. Video overlay information can then be inserted intodisplayed images, such as images displayed on one or more of thedisplays 206, 211. The overlay information may illustrate the physicalspace where accuracy of the 3D tracking system (which is typically partof the navigation system) is greater, may illustrate the available rangeof motion of the positioning system 208 and/or the imaging system 500,and/or may help to guide head and/or patient positioning.

The navigation system 205 may provide tools to the neurosurgeon that mayhelp to provide more relevant information to the surgeon, and may assistin improving performance and accuracy of port-based neurosurgicaloperations. Although described in the present disclosure in the contextof port-based neurosurgery (e.g., for removal of brain tumors and/or fortreatment of intracranial hemorrhages (ICH)), the navigation system 205may also be suitable for one or more of: brain biopsy,functional/deep-brain stimulation, catheter/shunt placement (in thebrain or elsewhere), open craniotomies, and/orendonasal/skull-based/ear-nose-throat (ENT) procedures, among others.The same navigation system 205 may be used for carrying out any or allof these procedures, with or without modification as appropriate.

For example, although the present disclosure may discuss the navigationsystem 205 in the context of neurosurgery, the same navigation system205 may be used to carry out a diagnostic procedure, such as brainbiopsy. A brain biopsy may involve the insertion of a thin needle into apatient's brain for purposes of removing a sample of brain tissue. Thebrain tissue may be subsequently assessed by a pathologist to determineif it is cancerous, for example. Brain biopsy procedures may beconducted with or without a stereotactic frame. Both types of proceduresmay be performed using image-guidance. Frameless biopsies, inparticular, may be conducted using the navigation system 205.

In some examples, the tracking camera 213 may be part of any suitabletracking system. In some examples, the tracking camera 213 (and anyassociated tracking system that uses the tracking camera 213) may bereplaced with any suitable tracking system which may or may not usecamera-based tracking techniques. For example, a tracking system thatdoes not use the tracking camera 213, such as a radiofrequency trackingsystem, may be used with the navigation system 205.

FIG. 3 is a block diagram illustrating a control and processing system300 that may be used in the medical navigation system 205 shown in FIG.2 (e.g., as part of the equipment tower 207). As shown in FIG. 3, in oneexample, control and processing system 300 may include one or moreprocessors 302, a memory 304, a system bus 306, one or more input/outputinterfaces 308, a communications interface 310, and storage device 312.The control and processing system 300 may be interfaced with otherexternal devices, such as a tracking system 321, data storage 342, andexternal user input and output devices 344, which may include, forexample, one or more of a display, keyboard, mouse, sensors attached tomedical equipment, foot pedal, and microphone and speaker. Data storage342 may be any suitable data storage device, such as a local or remotecomputing device (e.g. a computer, hard drive, digital media device, orserver) having a database stored thereon. In the example shown in FIG.3, data storage device 342 includes identification data 350 foridentifying one or more medical instruments 360 and configuration data352 that associates customized configuration parameters with one or moremedical instruments 360. The data storage device 342 may also includepreoperative image data 354 and/or medical procedure planning data 356.Although the data storage device 342 is shown as a single device in FIG.3, it will be understood that in other embodiments, the data storagedevice 342 may be provided as multiple storage devices.

The medical instruments 360 may be identifiable by the control andprocessing unit 300. The medical instruments 360 may be connected to andcontrolled by the control and processing unit 300, or the medicalinstruments 360 may be operated or otherwise employed independent of thecontrol and processing unit 300. The tracking system 321 may be employedto track one or more medical instruments 360 and spatially register theone or more tracked medical instruments to an intraoperative referenceframe. For example, the medical instruments 360 may include trackingmarkers such as tracking spheres that may be recognizable by thetracking camera 213. In one example, the tracking camera 213 may be aninfrared (IR) tracking camera. In another example, as sheath placed overa medical instrument 360 may be connected to and controlled by thecontrol and processing unit 300.

The control and processing unit 300 may also interface with a number ofconfigurable devices, and may intraoperatively reconfigure one or moreof such devices based on configuration parameters obtained fromconfiguration data 352. Examples of devices 320, as shown in FIG. 3,include one or more external imaging devices 322, one or moreillumination devices 324, the positioning system 208, the trackingcamera 213, one or more projection devices 328, and one or more displays206, 211.

Exemplary aspects of the disclosure can be implemented via theprocessor(s) 302 and/or memory 304. For example, the functionalitiesdescribed herein can be partially implemented via hardware logic in theprocessor 302 and partially using the instructions stored in the memory304, as one or more processing modules or engines 370. Exampleprocessing modules include, but are not limited to, a user interfaceengine 372, a tracking module 374, a motor controller 376, an imageprocessing engine 378, an image registration engine 380, a procedureplanning engine 382, a navigation engine 384, and a context analysismodule 386. While the example processing modules are shown separately inFIG. 3, in some examples the processing modules 370 may be stored in thememory 304 and the processing modules 370 may be collectively referredto as processing modules 370. In some examples, two or more modules 370may be used together to perform a function. Although depicted asseparate modules 370, the modules 370 may be embodied as a unified setof computer-readable instructions (e.g., stored in the memory 304)rather than distinct sets of instructions.

It is to be understood that the system is not intended to be limited tothe components shown in FIG. 3. One or more components of the controland processing system 300 may be provided as an external component ordevice. In one example, the navigation module 384 may be provided as anexternal navigation system that is integrated with the control andprocessing system 300.

Some embodiments may be implemented using the processor 302 withoutadditional instructions stored in memory 304. Some embodiments may beimplemented using the instructions stored in memory 304 for execution byone or more general purpose microprocessors. Thus, the disclosure is notlimited to a specific configuration of hardware and/or software.

In some examples, the navigation system 205, which may include thecontrol and processing unit 300, may provide tools to the surgeon thatmay help to improve the performance of the medical procedure and/orpost-operative outcomes. In addition to removal of brain tumours andintracranial hemorrhages (ICH), the navigation system 205 can also beapplied to a brain biopsy, a functional/deep-brain stimulation, acatheter/shunt placement procedure, open craniotomies,endonasal/skull-based/ENT, spine procedures, and other parts of the bodysuch as breast biopsies, liver biopsies, etc. While several exampleshave been provided, examples of the present disclosure may be applied toany suitable medical procedure.

FIG. 4A is a flow chart illustrating an example method 400 of performinga port-based surgical procedure using a navigation system, such as themedical navigation system 205 described in relation to FIGS. 1 and 2. Ata first block 402, the port-based surgical plan is imported.

Once the plan has been imported into the navigation system at the block402, the patient is affixed into position using a body holdingmechanism. The head position is also confirmed with the patient plan inthe navigation system (block 404), which in one example may beimplemented by the computer or controller forming part of the equipmenttower 207.

Next, registration of the patient is initiated (block 406). The phrase“registration” or “image registration” refers to the process oftransforming different sets of data into one coordinate system. Data mayinclude multiple photographs, data from different sensors, times,depths, or viewpoints. The process of “registration” is used in thepresent application for medical imaging in which images from differentimaging modalities are co-registered. Registration is used in order tobe able to compare or integrate the data obtained from these differentmodalities.

Those skilled in the relevant arts will appreciate that there arenumerous registration techniques available and one or more of thetechniques may be applied to the present example. Non-limiting examplesinclude intensity-based methods that compare intensity patterns inimages via correlation metrics, while feature-based methods findcorrespondence between image features such as points, lines, andcontours. Image registration methods may also be classified according tothe transformation models they use to relate the target image space tothe reference image space. Another classification can be made betweensingle-modality and multi-modality methods. Single-modality methodstypically register images in the same modality acquired by the samescanner or sensor type, for example, a series of magnetic resonance (MR)images may be co-registered, while multi-modality registration methodsare used to register images acquired by different scanner or sensortypes, for example in magnetic resonance imaging (MRI) and positronemission tomography (PET). In the present disclosure, multi-modalityregistration methods may be used in medical imaging of the head and/orbrain as images of a subject are frequently obtained from differentscanners. Examples include registration of brain computerized tomography(CT)/MRI images or PET/CT images for tumor localization, registration ofcontrast-enhanced CT images against non-contrast-enhanced CT images, andregistration of ultrasound and CT.

FIG. 4B is a flow chart illustrating an example method involved inregistration block 406 as outlined in FIG. 4A, in greater detail. If theuse of fiducial touch points (440) is contemplated, the method involvesfirst identifying fiducials on images (block 442), then touching thetouch points with a tracked instrument (block 444). Next, the navigationsystem computes the registration to reference markers (block 446).

Alternately, registration can also be completed by conducting a surfacescan procedure (block 450). The block 450 is presented to show analternative approach, but may not typically be used when using afiducial pointer. First, the face is scanned using a 3D scanner (block452). Next, the face surface is extracted from MR/CT data (block 454).Finally, surfaces are matched to determine registration data points(block 456).

Upon completion of either the fiducial touch points (440) or surfacescan (450) procedures, the data extracted is computed and used toconfirm registration at block 408, shown in FIG. 4A.

Referring back to FIG. 4A, once registration is confirmed (block 408),the patient is draped (block 410). Typically, draping involves coveringthe patient and surrounding areas with a sterile barrier to create andmaintain a sterile field during the surgical procedure. The purpose ofdraping is to eliminate the passage of microorganisms (e.g., bacteria)between non-sterile and sterile areas. At this point, conventionalnavigation systems require that the non-sterile patient reference isreplaced with a sterile patient reference of identical geometry locationand orientation.

Upon completion of draping (block 410), the patient engagement pointsare confirmed (block 412) and then the craniotomy is prepared andplanned (block 414).

Upon completion of the preparation and planning of the craniotomy (block414), the craniotomy is cut and a bone flap is temporarily removed fromthe skull to access the brain (block 416). Registration data is updatedwith the navigation system at this point (block 422).

Next, the engagement within craniotomy and the motion range areconfirmed (block 418). Next, the procedure advances to cutting the duraat the engagement points and identifying the sulcus (block 420).

Thereafter, the cannulation process is initiated (block 424).Cannulation involves inserting a port into the brain, typically along asulci path as identified at 420, along a trajectory plan. Cannulation istypically an iterative process that involves repeating the steps ofaligning the port on engagement and setting the planned trajectory(block 432) and then cannulating to the target depth (block 434) untilthe complete trajectory plan is executed (block 424).

Once cannulation is complete, the surgeon then performs resection (block426) to remove part of the brain and/or tumor of interest. The surgeonthen decannulates (block 428) by removing the port and any trackinginstruments from the brain. Finally, the surgeon closes the dura andcompletes the craniotomy (block 430). Some aspects of FIG. 4A arespecific to port-based surgery, such as portions of blocks 428, 432, and434, but the appropriate portions of these blocks may be skipped orsuitably modified when performing non-port based surgery.

When performing a surgical procedure using a medical navigation system205, as outlined in connection with FIGS. 4A and 4B, the medicalnavigation system 205 may acquire and maintain a reference of thelocation of the tools in use as well as the patient in three dimensional(3D) space. In other words, during a navigated neurosurgery, there maybe a tracked reference frame that is fixed (e.g., relative to thepatient's skull). During the registration phase of a navigatedneurosurgery (e.g., the step 406 shown in FIGS. 4A and 4B), atransformation is calculated that maps the frame of reference ofpreoperative MRI or CT imagery to the physical space of the surgery,specifically the patient's head. This may be accomplished by thenavigation system 205 tracking locations of fiducial markers fixed tothe patient's head, relative to the static patient reference frame. Thepatient reference frame is typically rigidly attached to the headfixation device, such as a Mayfield clamp. Registration is typicallyperformed before the sterile field has been established (e.g., the step410 shown in FIG. 4A).

FIG. 5 illustrates use of an example imaging system 500, describedfurther below, in a medical procedure. Although FIG. 5 shows the imagingsystem 500 being used in the context of a navigation system environment200 (e.g., using a navigation system as described above), the imagingsystem 500 may also be used outside of a navigation system environment(e.g., without any navigation support).

An operator, typically a surgeon 201, may use the imaging system 500 toobserve the surgical site (e.g., to look down an access port). Theimaging system 500 may be attached to a positioning system 208 (e.g., acontrollable and adjustable robotic arm). The position and orientationof the positioning system 208, imaging system 500 and/or access port maybe tracked using a tracking system, such as described for the navigationsystem 205. The distance d between the imaging system 500 (morespecifically, the aperture of the imaging system 500) and the viewingtarget (e.g., the surface of the surgical site) may be referred to asthe working distance. The imaging system 500 may be designed to be usedin a predefined range of working distance (e.g., in the range of about15 cm to about 75 cm). It should be noted that, if the imaging system500 is mounted on the positioning system 208, the actual available rangeof working distance may be dependent on both the working distance of theimaging system 500 as well as the workspace and kinematics of thepositioning system 208.

FIGS. 6 and 7 are perspective views of an example embodiment of theimaging system 500. In this example, the imaging system 500 is shownmounted to the positioning system 208 (e.g., a robotic arm) of anavigation system. The imaging system 500 is shown with a housing 555that encloses the zoom and focus optics, the zoom and focus actuators,the camera, the controller and the 3D scanner, discussed further belowwith reference to FIG. 8. The housing is provided with a frame 560 onwhich trackable markers may be mounted, to enable tracking by thenavigation system. The imaging system 500 communicates with thenavigation system via a cable 565 (shown partially cut off). The distalend of the imaging system 500 is provided with light sources 570. Theexample shows four broad spectrum LEDs, however more or less lightsources 570 may be used, of any suitable type. Although the lightsources 570 are shown provided surrounding the aperture 553 of theimaging system 500, in other examples the light source(s) 570 may belocated elsewhere on the imaging system 500. The distal end of theimaging system 500 may also include openings 575 for the cameras of theintegrated 3D scanner. A support connector 580 for mounting the imagingsystem 500 to the positioning system 208 is also shown, as well as theframe 560 for mounting trackable markers.

FIG. 8 is a block diagram showing components of an example imagingsystem 500, which may be a surgical microscope. The imaging system 500may include an optical assembly 505 (also referred to as an opticaltrain). The optical assembly 505 may include optics (e.g., lenses,optical fibers, etc.) for focusing and zooming on the viewing target.The optical assembly 505 may include zoom optics 510 (which may includeone or more zoom lenses) and focus optics 515 (which may include one ormore focus lenses). Each of the zoom optics 510 and focus optics 515 areindependently moveable within the optical assembly, in order to adjustthe zoom and focus, respectively. Where the zoom optics 510 and/or thefocus optics 515 include more than one lens, each individual lens may beindependently moveable. The optical assembly 505 may include an aperture(not shown), which may be adjustable.

The imaging system 500 may include a zoom actuator 520 and a focusactuator 525 for positioning the zoom optics 510 and the focus optics515, respectively. The zoom actuator 520 and/or the focus actuator 525may be an electric motor, or other types of actuators including, forexample, pneumatic actuators, hydraulic actuators, shape-changingmaterials (e.g., piezoelectric materials or other smart materials) orengines, among other possibilities. In some examples, the zoom actuator520 and/or the focus actuator 525 may be implemented using a steppermotor and string-pulley drive system, for example as described in USPat. Pub. No. 2006/0187562, the entirety of which is hereby incorporatedby reference.

Although the term “motorized” is used in the present disclosure, itshould be understood that the use of this term does not limit thepresent disclosure to use of motors necessarily, but is intended tocover all suitable actuators, including motors. Although the zoomactuator 520 and the focus actuator 525 are shown outside of the opticalassembly 505, in some examples the zoom actuator 520 and the focusactuator 525 may be part of or integrated with the optical assembly 505.The zoom actuator 520 and the focus actuator 525 may operateindependently, to control positioning of the zoom optics 510 and thefocus optics 515, respectively. The lens(es) of the zoom optics 510and/or the focus optics 515 may be each mounted on a linear stage (e.g.,a motion system that restricts an object to move in a single axis, whichmay include a linear guide and an actuator; or a conveyor system such asa conveyor belt mechanism) that is moved along a set of rails by thezoom actuator 520 and/or the focus actuator 525, respectively, tocontrol positioning of the zoom optics 510 and/or the focus optics 515.The independent operation of the zoom actuator 520 and the focusactuator 525 may enable the zoom and focus to be adjusted independently.Thus, when an image is in focus, the zoom may be adjusted withoutrequiring further adjustments to the focus optics 515 to produce afocused image.

Operation of the zoom actuator 520 and the focus actuator 525 may becontrolled by a controller 530 (e.g., a microprocessor) of the imagingsystem 500. The controller 530 may receive control input (e.g., from anexternal system, such as an external processor or an input device).Where the imaging system 500 is used as part of the navigation system205, the controller 530 may communicate with and receive control inputfrom a processor of the navigation system 205. The control input mayindicate a desired zoom and/or focus, and the controller 530 may inresponse control the zoom actuator 520 and/of focus actuator 525 to movethe zoom optics 510 and/or the focus optics 515 accordingly to achievethe desired zoom and/or focus. In some examples, the zoom optics 510and/or the focus optics 515 may be moved or actuated without the use ofthe zoom actuator 520 and/or the focus actuator 525. For example, thefocus optics 515 may use electrically-tunable lenses or other deformablematerial that may be controlled directly by the controller 530.

By providing the controller 530, the zoom actuator 520 and the focusactuator 525 all as part of the imaging system 500, the imaging system500 may enable an operator (e.g., a surgeon) to control zoom and/orfocus during a medical procedure without having to manually adjust thezoom and/or focus optics 510, 515. For example, the operator may providecontrol input to the controller 530 verbally (e.g., via a voicerecognition input system), by instructing an assistant to enter controlinput into an external input device (e.g., into a user interfaceprovided by a workstation), using a foot pedal, or by other such means.In some examples, the controller 530 may carry out preset instructionsto maintain the zoom and/or focus at preset values (e.g., to performautofocusing) without requiring further control input during the medicalprocedure.

As mentioned above, an external processor (e.g., a processor of aworkstation or the navigation system) in communication with thecontroller 530 may be used to provide control input to the controller530. For example, the external processor may provide a graphical userinterface via which the operator or an assistant may input instructionsto control zoom and/or focus of the imaging system 500. The controller530 may alternatively or additionally be in communication with anexternal input system (e.g., a voice recognition input system or a footpedal).

The optical assembly 505 may also include one or more auxiliary opticssuch as a filter wheel 540 for selecting an optical filter for imaging.The filter wheel 540 may hold one or more optical filters, for examplean optical filter for fluorescence imaging. The filter wheel 540 may beactuated by a filter wheel actuator 542, which may be controlled by thecontroller 530, to place a selected optical filter in the optical path.

The imaging system 500 may also include a camera 535 (e.g., ahigh-definition (HD) camera) that captures image data from the opticalassembly. Operation of the camera may be controlled by the controller530. The camera 535 may also output data to an external system (e.g., anexternal workstation or external output device) to view the capturedimage data. In some examples, the camera 535 may output data to thecontroller 530, which in turn transmits the data to an external systemfor viewing. By providing image data to an external system for viewing,the captured images may be viewed on a larger display and may bedisplayed together with other information relevant to the medicalprocedure, including navigational information (e.g., a wide-field viewof the surgical site, navigation markers, 3D images, etc.). Providingthe camera 535 with the imaging system 500 may help to improve theconsistency of image quality among different medical centers.

Image data captured by the camera 535 may be displayed on a displaytogether with a wide-field view of the surgical site, for example in amultiple-view user interface. The portion of the surgical site that iscaptured by the camera 535 may be visually indicated in the wide-fieldview of the surgical site.

The imaging system 500 may include a three-dimensional (3D) scanner 545or 3D camera for obtaining 3D information of the viewing target. 3Dinformation from the 3D scanner 545 may also be captured by the camera535, or may be captured by the 3D scanner 545 itself. Operation of the3D scanner 545 may be controlled by the controller 530, and the 3Dscanner 545 may transmit data to the controller 530. In some examples,the 3D scanner 545 may itself transmit data to an external system (e.g.,an external work station). 3D information from the 3D scanner 545 may beused to generate a 3D image of the viewing target (e.g., a 3D image of atarget tumor to be resected). 3D information may also be useful in anaugmented reality (AR) display provided by an external system. Forexample an AR display (e.g., provided via AR glasses) may, usinginformation from a navigation system to register 3D information withoptical images, overlay a 3D image of a target specimen on a real-timeoptical image (e.g., an optical image captured by the camera 535).

The controller 530 may be coupled to a memory 550. The memory 550 may beinternal or external of the imaging system 500. Data received by thecontroller 530 (e.g., image data from the camera 535 and/or 3D data fromthe 3D scanner) may be stored in the memory 550. The memory 550 may alsocontain instructions to enable the controller to operate the zoomactuator 520 and the focus actuator 525. For example, the memory 550 maystore instructions to enable the controller to control the actuators520, 525 according to different control parameters, as discussed furtherbelow.

The imaging system 500 may communicate with an external system (e.g., anavigation system or a workstation) via wired or wireless communication.In some examples, the imaging system 500 may include a wirelesstransceiver (not shown) to enable wireless communication.

In some examples, the imaging system 500 may include a power source(e.g., a battery) or a connector to a power source (e.g., an ACadaptor). In some examples, the imaging system 500 may receive power viaa connection to an external system (e.g., an external workstation orprocessor).

In some examples, the imaging system 500 may include a light source (notshown). In some examples, the light source may not itself generate lightbut rather direct light from another light generating component. Forexample, the light source may be an output of a fibre optics cableconnected to another light generating component, which may be part ofthe imaging system 500 or external to the imaging system 500. The lightsource may be mounted near the aperture of the optical assembly, todirect light to the viewing target. Providing the light source with theimaging system 500 may help to improve the consistency of image qualityamong different medical centers. In some examples, the power or outputof the light source may be controlled by the imaging system 500 (e.g.,by the controller 530) or may be controlled by a system external to theimaging system 500 (e.g., by an external workstation or processor, suchas a processor of a navigation system).

In some examples, the optical assembly 505, zoom actuator 520, focusactuator 525 and camera 535 may all be housed within a single housing ofthe imaging system 500. In some examples, the controller 530, memory550, 3D scanner 545, wireless transceiver, power source and/or lightsource may also be housed within the housing.

In some examples, the imaging system 500 may also provide mechanisms toenable manual adjusting of the zoom and/or focus optics 510, 515,similarly to conventional systems. Such manual adjusting may be enabledin addition to motorized adjusting of zoom and focus. In some examples,such manual adjusting may be enabled in response to user selection of a“manual mode” on a user interface.

The imaging system 500 may be mountable on a moveable support structure,such as the positioning system (e.g., robotic arm) of a navigationsystem, a manually operated support arm, a ceiling mounted support, amoveable frame, or other such support structure. The imaging system 500may be removably mounted on the moveable support structure. In someexamples, the imaging system 500 may include a support connector (e.g.,a mechanical coupling) to enable the imaging system 500 to be quicklyand easily mounted or dismounted from the support structure. The supportconnector on the imaging system 500 may be configured to be suitable forconnecting with a typical complementary connector on the supportstructure (e.g., as designed for typical end effectors). In someexamples, the imaging system 500 may be mounted to the support structuretogether with other end effectors, or may be mounted to the supportstructure via another end effector.

When mounted, the imaging system 500 may be at a known fixed positionand orientation relative to the support structure (e.g., by calibratingthe position and orientation of the imaging system 500 after mounting).In this way, by determining the position and orientation of the supportstructure (e.g., using a navigation system or by tracking the movementof the support structure from a known starting point), the position andorientation of the imaging system 500 may also be determined. In someexamples, the imaging system 500 may include a manual release buttonthat, when actuated, enable the imaging system 500 to be manuallypositioned (e.g., without software control by the support structure).

In some examples, where the imaging system 500 is intended to be used ina navigation system environment, the imaging system 500 may include anarray of trackable markers, which may be mounted on a frame on theimaging system 500) to enable the navigation system to track theposition and orientation of the imaging system 500. Alternatively oradditionally, the moveable support structure (e.g., a positioning systemof the navigation system) on which the imaging system 500 is mounted maybe tracked by the navigation system and the position and orientation ofthe imaging system 500 may be determined using the known position andorientation of the imaging system 500 relative to the moveable supportstructure.

The trackable markers may include passive reflective tracking spheres,active infrared (IR) markers, active light emitting diodes (LEDs), agraphical pattern, or a combination thereof. There may be at least threetrackable markers provided on a frame to enable tracking of position andorientation. In some examples, there may be four passive reflectivetracking spheres coupled to the frame. While some specific examples ofthe type and number of trackable markers have been given, any suitabletrackable marker and configuration may be used, as appropriate.

Determination of the position and orientation of the imaging system 500relative to the viewing target may be performed by a processor externalto the imaging system 500 (e.g., a processor of the navigation system).Information about the position and orientation of the imaging system 500may be used, together with a robotic positioning system, to maintainalignment of the imaging system 500 with the viewing target (e.g., toview down an access port during port-based surgery) throughout themedical procedure.

For example, the navigation system may track the position andorientation of the positioning system and/or the imaging system 500either collectively or independently. Using this information as well astracking of the access port, the navigation system may determine thedesired joint positions for the positioning system so as to maneuver theimaging system 500 to the appropriate position and orientation tomaintain alignment with the viewing target (e.g., the longitudinal axesof the imaging system 500 and the access port being aligned). Thisalignment may be maintained throughout the medical procedureautomatically, without requiring explicit control input. In someexamples, the operator may be able to manually move the positioningsystem and/or the imaging system 500 (e.g., after actuation of a manualrelease button). During such manual movement, the navigation system maycontinue to track the position and orientation of the positioning systemand/or the imaging system 500. After completion of manual movement, thenavigation system may (e.g., in response to user input, such as using afoot pedal, indicating that manual movement is complete) reposition andreorient the positioning system and the imaging system 500 to regainalignment with the access port.

The working distance may be determined by the controller 530 usinginformation (e.g., received from the navigation system, from thepositioning system or other external system) about the position andorientation of the imaging system 500 and/or the positioning systemrelative to the viewing target. In some examples, the working distancemay be determined by the controller 530 using an infrared light (notshown) mounted on near the distal end of the imaging system 500.

In some examples, the mechanism for moving the zoom optics may be astepper motor and string-pulley drive system. Using such a drive system,it may not be practical or possible to implement a gear ratio that isadequate for high precision and high accuracy in control, when a highzoom level is used. Backlash in the gearbox and insufficiency in theholding force may also result in movement of the optical image, causingimage instability, which may also be variable with the orientation ofthe imaging system. The pulley drive mechanism may be sensitive tobacklash, which may prevent or hinder use of the drive system for finemovement at a high zoom level. As well, there may be variation infriction along the rail on which the zoom optics are moved.

FIG. 9 illustrates an example control loop 900 that may be used by thecontroller 530 to control the zoom optics 510 and/or the focus optics515. For the purpose of generalization, FIG. 9 illustrates control forgeneral optics 910 that are positioned by a general actuator 908. Itshould be understood that the actuator 908 may be the zoom actuator 520and/or the focus actuator 525, and the optics 910 may be the zoom optics510 and/or the focus optics 515. A single instance of the control loop900 may be used to control a single set of optics 910. Thus, there maybe multiple instances of the control loop 900. For example, there may bethree such control loops 900 implemented by the controller 530—onecontrol loop 900 to control the focus optics 515, and two control loops900 to control two zoom optics 510. The control loops 900 may beimplemented using a master-slave configuration, in which the controller530 includes a master controller and each set of optics 910 iscontrolled by a respective slave controller.

A single instance of the control loop 900 is now described. The targetposition 902 (e.g., received as control input into the mastercontroller) is inputted into the position control 904 (which may beimplemented by the master controller, where a master-slave configurationis used), which implements a proportional-integral-derivative (PID)control loop, represented by the transfer function (in the Laplacedomain) Kp+Ki/s+Kd*s, where s is the complex frequency and the controlterms are Kp (proportional), Ki (integral) and Kd (derivative). Theactual monitored position of the optics 910 is a negative feedback tothe position control 904. The output of the position control 904 isinputted to the actuator control 906 (which may be implemented by aslave controller, where a master-slave configuration is used), whichcontrols the current of the actuator 908 (e.g., brushless DC (BLDC)motor). The actuator control 906 implements a proportional-integral (PI)control loop, represented by the transfer function (in the Laplacedomain) Kp+Ki/s. The actual current of the actuator 908 is a negativefeedback to the actuator control 906.

In some conventional surgical microscopes, at steady state theconventional system sometimes exhibits a state of instability, resultingin oscillations of the image. This may be due to variations in frictionand/or stiction in the physical movement of the optics, and may also bedue to backlash in the gearing of the actuation motor. Although suchinstability may be mitigated by using less aggressive position controlparameters, the trade-off is a poorer step response time, which may notsatisfy user requirements.

In the disclosed imaging system, instead of using a single set ofcontrol parameters, the controller 530 controls each instance of thecontrol loop 900 according to two sets of control parameters, in thiscase two sets of PID parameters thus implementing a dual-PID control.Such a configuration may help to mitigate the jitter and instabilitychallenges discussed above. PID controls are typically used to controlsystems with a defined input (in this case a target position 902) and anoutput (in this case the actual position of the optics 910) that can bemonitored.

FIG. 10 is a flowchart illustrating an example method 1000 forcontrolling the optics 910, according to two sets of control parameters.The method 1000 may be performed by the controller 530 to control eachset of optics (e.g., each set of focus optics and zoom optics)independently. Each set of optics may have their own respective firstand second sets of control parameters.

At 1002, a new target position 902 is received into the control loop900, to change the position of the optics 910 from the current steadystate.

At 1004, the controller 530 implements a first set of controlparameters, which may be referred to as coarse control parameters K(a).The coarse control parameters K(a) are designed to be more aggressive,to move the optics 910 more quickly. The position of the optics 910 maybe detected by a sensor (e.g., opto-electrical sensor) in the imagingsystem 500, or otherwise tracked or determined by the controller 530.

At 1006, when the optics 910 are within a tolerance or threshold range(e.g., within 10 μm) of the target position 902 (e.g., for at least athreshold amount of time), the controller 530 switches to a second setof control parameters, which may be referred to as fine controlparameters K(c). The fine control parameters K(c) are designed to bemore conservative, to move the optics 910 more carefully and slowly. Thesecond set of control parameters K(c) may have Kp(c) and Kd(c) controlterms similar to the Kp(a) and Kd(a) control terms of the first set ofcontrol parameters K(a), but may have a smaller Ki(c) control term(e.g., by a factor of about 2 to 3) than the Ki(a) control term of thefirst set of control parameters K(a). By using a smaller Ki(c) term, thelong-term effect of small deviations in position is reduced.

In some examples, the controller 530 may also switch from the first setof control parameters K(a) to the second set of control parameters K(c)upon expiry of a preset timer, regardless of the position of the optics910. This may be a fail-safe mechanism.

At 1008, when the optics 910 reaches the target position 902, thecontroller 530 maintains the optics 910 at steady state at the targetposition 902, using the second set of control parameters K(c), until anew target position 902 is received.

In some examples, the controller 530 may also implement a clamp on theintegrated error sum to prevent integrator windup.

Using the example method 1000, the imaging system 500 may achieve a faststep response time (e.g., negligible lag between user input and changein focus and/or zoom) while achieving a position accuracy for the optics910 of 10 μm or less at steady state (i.e., when holding a desired focusand/or zoom).

The control terms for the first and second set of control parametersK(a), K(c) may be individually adjusted or otherwise defined for eachindividual imaging system 500 (e.g., at the manufacturing stage), inorder to achieve the desired level of stability and in order to accountfor individual variations (e.g., variations in rail friction for eachimaging system).

The control parameters K(a), K(c) may be tuned using any suitabletechniques, for example using a modified version of the heuristicZiegler and Nichols method. FIG. 11 is a flowchart illustrating anexample tuning method 1100 for the control parameters K(a), K(c). Themethod 1100 may be performed by the controller 530 of the imaging system500, or by an external controller.

Tuning of the first set of control parameters K(a), which is the moreaggressive set of parameters, is performed first at 1102.

At 1104, all control terms of K(a) are initiated with zero gains.

At 1106, Kp(a) is incremented until the system exhibits a sustainedoscillation due to a unit step.

At 1108, set Kp(a)=Kp(a)/2. In some examples, a factor other than twomay be used, as appropriate.

At 1110, Kd(a) is incremented until the system achieves a slightlyover-damped step response. The typical ratio of Kd(a):Kp(a) thatachieved this result has been found to be about 4:1, for the exampleimaging system driven using a pulley system as described above.

At 1112, Ki(a) is incremented until the system achieves the desiredpositioning accuracy. The ratio of Ki(a) to the other control terms wasfound to vary among individual systems.

Tuning of the second set of control parameters K(c), which is the moreconservative set of parameters, is performed next at 1114, based on thefirst set of control parameters K(a).

At 1116, set Ki(c)=Ki(a)/3. In some examples, instead of a reduction bya factor of three, a reduction by another factor, for example 2 or someother amount in a similar range, may be used.

At 1118, temporarily set Ki(c)=Ki(c)*10 and input a unit step to inducesystem instability.

At 1120, apply the control parameters K(c). If the system does notreturn to a stable state, Ki(c) is decreased (e.g., by intervals of 0.1)until stability is recovered.

At 1122, the tuned control parameters K(a) and K(c) are stored inmemory, to be accessed by the controller 530 when needed (e.g., for themethod 1000).

The method 1100 may be performed for each individual imaging system 500,and may be performed by the manufacturer. The method 1100 may beperformed for each set of optics (e.g., each set of focus optics andzoom optics) to obtain control parameters for each set of opticsindependently. The method 1100 may also be performed any time during thelifetime of the imaging system 500, for example to adjust the controlparameters for greater stability requirements and/or to adjust thecontrol parameters for changes in the physical properties of the imagingsystem 500 (e.g., changes in friction of the actuator, due to wear andtear over the lifetime of the imaging system 500).

Although two sets of control parameters, K(a) and K(c), have beendiscussed, in some examples there may be more than two sets of controlparameters. For example, there could be multiple sets of controlparameters that can be selectively used for different operationconditions. In some examples, the control parameters may be a functionof one or more other variables or parameters. For example, the controlloop 900 may use a set of control parameters that is a function of thezoom level (e.g., the control parameters may be less aggressive at ahigher zoom level and more aggressive at a lower zoom level).

In some examples, there may be a misalignment between the mechanicalaxis of the imaging system and its optical axis. An example of such amisalignment is illustrated in FIG. 12, with dimensions exaggerated forclarity. FIG. 12 illustrates an imaging system 500 in which the housing555 has a longitudinal axis defining a mechanical axis M of the imagingsystem. The optical train 556 of the imaging system 500 has an opticalaxis O, which is the axis for capturing an optical image. Although shownas an exaggerated misalignment in one degree of freedom, typically themisalignment is small (e.g., about 0.15° to about 0.25°, in someexamples no more than a maximum of 4°) and may be in more than onedegree of freedom (e.g., may be translational and/or rotational alongand/or around the x, y and/or z-axes). In typical applications, suchmisalignment may not be noticeable. However, in surgical microscopeapplications, due to the high zoom level required together with therelatively large working distance, even a misalignment of 0.1° may beunacceptable.

As described above, the imaging system 500 may be supported by apositioning system, such as a robotic arm, and the position andorientation of the imaging system 500 may be changed by moving thepositioning system. The position and orientation of the imaging system500 may be tracked by a tracking system that tracks tracking markerscoupled to the housing 555 of the imaging system 500 and/or trackingmarkers coupled to the robotic arm. The positioning of the imagingsystem 500 may be based on the assumption that the optical axis O isaligned with the mechanical axis M. For example, to increase or decreasethe working distance of the imaging system 500, the imaging system 500may be moved along the mechanical axis M. However, if there ismisalignment between the optical axis O and the mechanical axis M, thenwhen the imaging system 500 is moved along the mechanical axis M, thecenter point of the captured image with shift, resulting in a shift inthe field-of-view (FOV) of the captured image. This shift is undesirableand unexpected for the surgeon. This FOV error may be significant whenusing the high zoom level and long working distance required forsurgical microscopes during a surgical procedure.

Table 1 below illustrates the FOV error (as percentage of FOV) that mayarise for example angular misalignment that may be found in typicalimaging systems, at various working distances and zoom levels that maybe used during a surgical procedure.

Angular misalignment Working distance (mm) Zoom (×) 4° 0.15° 0.26° 0.20°200 1  17%  <1%  1%  <1% 650 1  22%  <1%  1%    1% 200 12.5 215%    1%14%   10% 650 12.5 274%   10% 18%   14%

To compensate for such misalignment, a calibration of the optical axismay be performed. Using the example calibration technique describedbelow, a transformation may be calculated, which may be used tocompensate for misalignment of the optical axis, including translationalas well as rotational misalignment in six degrees of freedom. An exampleof a calibration apparatus 1300 for calibrating the optical axis isshown in FIG. 13. The calibration apparatus 1300 may be used forperforming the example calibration method 1400 illustrated in FIG. 14.FIGS. 13 and 14 will be described together.

Calibration of the optical axis may be performed using the trackingsystem together with video processing of the captured image. Thecalibration may be performed using the processor of the navigationsystem 205, for example, as part of setting up of the navigation system205 prior to the start of a medical procedure. In some examples,calibration of the optical axis may be performed by the manufacturer.The optical axis may be calibrated at the start of every medicalprocedure, during the medical procedure, or only at defined timeintervals (e.g., weekly), for example.

The calibration apparatus 1300 has a body 1302 in which is defined anupper feature (in this example an upper bore hole 1304) and a lowerfeature (in this example a divot 1306) that are aligned longitudinallyand separated by a known distance d_(c).

At 1402, the optical system 500 is positioned to have its mechanicalaxis aligned with the longitudinal axis of the calibration apparatus1300, and at a maximum working distance and maximum zoom level from thelower divot 1306. The optical system 500 is thus positioned to capturean image of the lower divot 1306.

At 1404, the focus of the imaging system 500 is adjusted to bring thelower feature (e.g. divot) 1306 into focus. This focus adjustment is thez-translation error. A first image of the lower feature (e.g. divot)1306 is captured.

At 1406, optionally, the zoom level of the imaging system 500 may bedecreased to perform roll error calibration. In some examples, it maynot be necessary to correct for roll error, because typically theimaging system 500 may be manufactured such that the roll error isalready small enough and does not require correction. If correction ofroll error is required, the roll error may be calculated by measuringthe angular amount by which a horizontal or vertical line in the imagedeviates from the actual horizontal or vertical of the image.

At 1408, the focus of the imaging system 500 is adjusted by the knowndistance d_(c), to capture a second image of the upper feature (e.g.bore hole) 1304.

At 1410, the processor (e.g., processor of the navigation system 205, anonboard processor of the imaging system 500, or another externalprocessor) determines the centroids of the upper feature (e.g. borehole) 1304 and lower feature (e.g. divot) 1306 captured in the first andsecond images. The x-y distance from the calculated centroid of thelower feature (e.g. divot) 1306 and the actual center of the image isthe x-y translation error. The x-y distance between the calculatedcentroid of the lower feature (e.g. divot) 1306 and the calculatedcentroid of the upper feature (e.g. bore hole) is the pitch-yaw error.An example calculation is shown below:

$\theta_{x} = {\tan^{- 1}\left( \frac{x_{shift}*\frac{mm}{pixel}}{d_{c}} \right)}$

where x_(shift) is the x-axis offset between the calculated centroids,mm/pixel is the known ratio of actual distance (mm) to each pixel of theimage, d_(c) is the known distance between the bore hole 1304 and divot1306, and θ_(x) is the angular error.

At 1412, the determined errors are used to apply corrections to theoptical axis matrix, to generate the correction terms for a correctionmatrix.

At 1414, the correction matrix is stored (e.g., by the processor thatperformed 1410). The correction matrix may thus be a transformationbetween the mechanical axis frame of reference and the optical axisframe of reference. Using the correction matrix, a desired increase ordecrease of the working distance along the optical axis may betransformed to the necessary change in x/y/z and pitch/yaw/roll of thepositioning system along the mechanical axis.

When a control input is received to move to a desired working distance,the processor (e.g., controller of the navigation system) may retrievethe correction matrix from memory and apply the correction matrix totransform the working distance to the mechanical axis frame ofreference. The positioning system may then be controlled according tothe transformed position and orientation.

In some examples, the calibration method 1400 may be repeated one ormore times, to achieve a desired amount of correction (e.g., <0.1°misalignment). In some examples, the calibration apparatus 1300 may haveother calibration points that may be used by the user to manually verifyaccuracy of the calibration.

Other calibration apparatuses may be used. For example, a differentcalibration target may use tracking markers for as calibration targets,or a crosshair instead of using upper and lower features (e.g. bore holeand divot, respectively).

In some examples, the imaging system 500 may include one or moreselectable optical filters. For example, as shown in FIG. 8, the imagingsystem 500 may include a filter wheel 540 holding one or more opticalfilters. The filter wheel 540 may be driven by a filter wheel actuator542, for example a stepper system that may be controlled by thecontroller 530 using an open-loop control. The filter wheel 540 may bepositioned to place a selected optical filter in the optical path. Oneof the optical filters may enable the imaging system 500 to captureimages in a fluorescence mode. Different optical filters may havedifferent coatings that alter the optical path, for example due torefraction of light as light enters and exits the optical filter. Anexample of this is illustrated in FIG. 15.

FIG. 15 illustrates the shift in optical path caused by an opticalfilter 1502. Light from a target 1504 is refracted at each side of theoptical filter 1502 (e.g., due to the coating on the optical filter1502), such that the optical path 1506 is shifted. When the light isobserved by a viewer 1508 (represented here as a human eye, but could beany light sensor, including any sensor or camera of the imaging system500), the target 1504 is perceived to be at a position 1510 that islaterally shifted relative to the actual position of the target 1504.For the imaging system 500, this causes a shift in focal point which, ifnot compensated, requires the focus to be manually adjusted each timethe optical filter 1502 is added, changed or removed, which can beburdensome and time-consuming.

In some examples, changes in temperature may also cause shift in thefocal point, for example due to thermal characteristics of the optics.At a first temperature (e.g., 15° C.), the focus optics may bepositioned at a first position x1 to focus on an object at a distance of25 cm, and positioned at a second position x2 to focus on an object at adistance of 35 cm; at a different second temperature (e.g., 30° C.), thefocus optics may need to be positioned at a third position x1+Δx1 tofocus on an object at a distance of 25 cm, and need to be positioned ata fourth position x2+Δx2 to focus on an object at a distance of 35 cm.

The optical filter-related and temperature-related shifts of the focalpoint may be considered negligible or within acceptable tolerance inmost optical applications. However, for a surgical microscope, operatingat high zoom level and long working distance, even small shifts (e.g.,on the order of 50 μm) may be noticeable and may cause unwanteddisruption to the medical procedure. It may be necessary for the surgeonto interrupt the procedure to refocus the image, to ensure accuracy andprecision of the procedure. This can be time-consuming and burdensome.

In some examples, the imaging system 500 may automatically compensatefor such optical filter-related and temperature-related shifts. Theoptical properties of the imaging system 500 may be calibrated (e.g.,during manufacturing, or at regular calibrations sessions) by empiricaltesting and/or simulations to determine the offset or shift caused bychanges in temperature and/or changes in optical filter. In some cases,this offset may be represented by a polynomial (e.g., representingoptical offset as a function of temperature). The offset may be specificto each individual imaging system 500 (e.g., dependent on specificoptical design and the optical filters used). The offset determined bythis calibration may be stored in a look-up table and/or referencepolynomial (e.g., in a memory internal to the imaging system 500) andmay be retrieved by the controller 530 of the imaging system 500 toperform compensation.

FIG. 16 is a flowchart illustrating an example method 1600 that may beperformed by the controller 530 of the imaging system 500 to compensatefor temperature-related and/or optical filter-related shifts.

At 1602, the controller 530 may receive a control input to focus on atarget at a certain distance. For example, the controller 530 mayreceive control input from the navigation system 205 to focus on atarget selected by a tracked pointing tool (e.g., as described inPCT/CA2015/050948, the entirety of which is hereby incorporated byreference).

At 1604, the controller 530 receives information from a temperaturesensor (which may be internal to the imaging system 500) indicating thesurrounding temperature of the optical assembly (or more specificallythe optics). The controller 530 also receives information or determinesif there is an optical filter being used and the type of optical filterbeing used.

At 1606, the controller 530 determines compensation amount by which theposition of the optics (e.g., focus optics) should be adjusted tocompensate for temperature-related and/or optical. The compensationamount may be determined by retrieving calibration data from a look-uptable and/or by calculating using a reference polynomial.

At 1608, the controller 530 adjusts the target position of the optics bythe compensation amount.

At 1610, the controller 530 controls the optics actuator (e.g., usingthe dual-PID control described above) to position the optics at theadjusted target position.

Although described above with respect to compensating for shifts infocal points by adjusting the position of the focus optics, a similarcalibration and compensation may be carried out to compensate foroptical filter-related and/or temperature-related shifts in the zoomoptics, or other optics of the imaging system 500.

The example methods 1000, 1100, 1400, 1600 described above may beentirely performed by the controller of the imaging system, or may bepartly performed by the controller and partly performed by an externalsystem. For example, one or more of: determining theposition/orientation of the imaging system, determining theposition/orientation of the imaging target or medical instrument,determining the working distance, or determining the desired position ofthe focus optics may be performed by one or more external systems. Thecontroller of the imaging system may simply receive commands, from theexternal system(s) to position the focus optics at the desired position,or the controller of the imaging system may determine the desiredposition of the focus optics after receiving the calculated workingdistance from the external system(s).

In various examples disclosed herein, methods, apparatuses and systemsare described which may help to address the intensive opticalrequirements of a large variable focal distance and large variable zoomdistance, for a compact, arm-mounted imaging system, such as a surgicalmicroscope. Such an imaging system provides specific challenges incontrolling the accuracy of the position of the optics. Conventionalmechanical systems (e.g., pulley system) for most imaging systems may becontrolled using a single set of control parameters, since thepositioning errors and/or jitter are typically no more than on the orderof micrometers, which is typically acceptable for most applications.However, in surgical microscopes, positioning errors and/or jitter onthe order of micrometers are not acceptable. Hence, the need for moreprecise and accurate control, which is addressed by examples disclosedherein, is unique to the surgical microscope application.

The need for optical axis calibration is also unique to the surgicalmicroscope application because the high magnification of the trackedimaging system leads to the manifestation of this error. At lower zoomlevels, the optical axis misalignment is not noticeable.

While some embodiments or aspects of the present disclosure may beimplemented in fully functioning computers and computer systems, otherembodiments or aspects may be capable of being distributed as acomputing product in a variety of forms and may be capable of beingapplied regardless of the particular type of machine or computerreadable media used to actually effect the distribution.

At least some aspects disclosed may be embodied, at least in part, insoftware. That is, some disclosed techniques and methods may be carriedout in a computer system or other data processing system in response toits processor, such as a microprocessor, executing sequences ofinstructions contained in a memory, such as read only memory (ROM),volatile random access memory (RAM), non-volatile memory, cache or aremote storage device.

A computer readable storage medium may be used to store software anddata which when executed by a data processing system causes the systemto perform various methods or techniques of the present disclosure. Theexecutable software and data may be stored in various places includingfor example ROM, volatile RAM, non-volatile memory and/or cache.Portions of this software and/or data may be stored in any one of thesestorage devices.

Examples of computer-readable storage media may include, but are notlimited to, recordable and non-recordable type media such as volatileand non-volatile memory devices, ROM, RAM, flash memory devices, floppyand other removable disks, magnetic disk storage media, optical storagemedia (e.g., compact discs (CDs), digital versatile disks (DVDs), etc.),among others. The instructions can be embodied in digital and analogcommunication links for electrical, optical, acoustical or other formsof propagated signals, such as carrier waves, infrared signals, digitalsignals, and the like. The storage medium may be the internet cloud, ora computer readable storage medium such as a disc.

Furthermore, at least some of the methods described herein may becapable of being distributed in a computer program product comprising acomputer readable medium that bears computer usable instructions forexecution by one or more processors, to perform aspects of the methodsdescribed. The medium may be provided in various forms such as, but notlimited to, one or more diskettes, compact disks, tapes, chips, USBkeys, external hard drives, wire-line transmissions, satellitetransmissions, internet transmissions or downloads, magnetic andelectronic storage media, digital and analog signals, and the like. Thecomputer useable instructions may also be in various forms, includingcompiled and non-compiled code.

At least some of the elements of the systems described herein may beimplemented by software, or a combination of software and hardware.Elements of the system that are implemented via software may be writtenin a high-level procedural language such as any suitable programming orscripting language. Accordingly, the program code may be written in C,C++, J++, or any other suitable programming language and may comprisemodules or classes, as is known to those skilled in programming. Atleast some of the elements of the system that are implemented viasoftware may be written in assembly language, machine language orfirmware as needed. In either case, the program code can be stored onstorage media or on a computer readable medium that is readable by ageneral or special purpose programmable computing device having aprocessor, an operating system and the associated hardware and softwarethat is necessary to implement the functionality of at least one of theembodiments described herein. The program code, when read by thecomputing device, configures the computing device to operate in a new,specific and predefined manner in order to perform at least one of themethods described herein.

While the teachings described herein are in conjunction with variousembodiments for illustrative purposes, it is not intended that theteachings be limited to such embodiments. On the contrary, the teachingsdescribed and illustrated herein encompass various alternatives,modifications, and equivalents, without departing from the describedembodiments, the general scope of which is defined in the appendedclaims. Except to the extent necessary or inherent in the processesthemselves, no particular order to steps or stages of methods orprocesses described in this disclosure is intended or implied. In manycases the order of process steps may be varied without changing thepurpose, effect, or import of the methods described.

1. An optical camera module comprising: an optical assembly including atleast one moveable optics; an actuator for positioning the at least onemoveable optics, the actuator including a pulley system for moving theat least one moveable optics; a camera for capturing an image from theoptical assembly; and an auxiliary optics holding multiple opticalfilters, the auxiliary optics being positionable to position one of themultiple optical filters in an optical path of the at least one moveableoptics; wherein the actuator is controlled to: position the at least onemoveable optics towards a target position according to a first set ofcontrol parameters when the at least one moveable optics is outside athreshold range of the target position; and maintain the at least onemoveable optics at the target position at steady state according to asecond set of control parameters when the moveable optics is within thethreshold range of the target position.
 2. The optical camera module ofclaim 1, wherein the multiple selectable optical filters comprise atleast one optical filter for fluorescence imaging.
 3. The optical cameramodule of claim 1, wherein the multiple optical filters comprise atleast one optical filter having a coating that alters the optical path.4. The optical camera module of claim 1, wherein the first and secondsets of control parameters are, respectively, first and second sets ofproportional (P), integral (I) and derivative (D) terms for dual-PIDcontrol of the actuator.
 5. The optical camera module of claim 4,wherein the second set of control parameters has an I term smaller thanthe I term of the first set of control parameters by a factor in therange of about 2 to
 3. 6. The optical camera module of claim 1, furthercomprising a controller configured to: control the actuator to positionthe at least one moveable optics towards the target position accordingto the first set of control parameters; and in response to receivingsignals from a sensor detecting that the at least one moveable optics iswithin the threshold range of the target position, switch to the secondset of control parameters to control the actuator to maintain the atleast one moveable optics at the target position at steady state.
 7. Theoptical camera module of claim 6, wherein the controller is furtherconfigured to: adjust the target position by a compensation amount tocompensate for at least one of an optical filter-related optical shiftor a temperature-related optical shift; and control the actuatoraccording to the adjusted target position.
 8. The optical camera moduleof claim 7, wherein the controller is configured to determine thecompensation amount using a look-up table or a reference polynomial. 9.The optical camera module of claim 1, wherein: the at least one moveableoptics includes moveable zoom optics and moveable focus optics; theoptical camera module comprises at least two actuators including a zoomactuator for positioning the zoom optics and a focus actuator forpositioning the focus optics; the zoom actuator and the focus actuatorbeing controlled independently according to first and second sets ofzoom control parameters and first and second sets of focus controlparameters, respectively.
 10. The optical camera module of claim 1,wherein the optical camera module is configured to be removablymountable to a moveable support structure.
 11. The optical camera moduleof claim 10, wherein the moveable support structure is a robotic arm.12. The optical camera module of claim 10, wherein the moveable supportstructure is a ceiling mounted support.
 13. The optical camera module ofclaim 1, for use in a surgical microscope for capturing a high opticalzoom image of a target during surgical procedure.
 14. A method forcontrolling an optical camera module, the method comprising: controllingan actuator including a pulley system to position at least one moveableoptics of the optical camera module towards a target position, theactuator being controlled according to a first set of controlparameters, when the at least one moveable optics is outside a thresholdrange of the target position; controlling the actuator to maintain theat least one moveable optics at the target position at steady state, theactuator being controlled according to a second set of controlparameters, when the at least one moveable optics is within thethreshold range of the target position; and controlling an auxiliaryoptics holding multiple optical filters to position one of the multipleoptical filters in an optical path of the at least one moveable optics.15. The method of claim 14, wherein the multiple selectable opticalfilters comprise at least one optical filter for fluorescence imaging,and controlling the auxiliary optics comprises controlling the auxiliaryoptics to position the at least one optical filter for fluorescenceimaging in the optical path.
 16. The method of claim 14, wherein thefirst and second sets of control parameters are, respectively, first andsecond sets of proportional (P), integral (I) and derivative (D) termsfor dual-PID control of the actuator.
 17. The method of claim 16,wherein the second set of control parameters has an I term smaller thanthe I term of the first set of control parameters by a factor in therange of about 2 to
 3. 18. The method of claim 14, further comprising:adjusting the target position by a compensation amount to compensate forat least one of an optical filter-related optical shift or atemperature-related optical shift; and controlling the actuatoraccording to the adjusted target position.
 19. The method of claim 14,wherein the at least one moveable optics includes moveable zoom opticsand moveable focus optics; wherein the actuator comprises at least twoactuators including a zoom actuator for positioning the zoom optics anda focus actuator for positioning the focus optics; wherein the first setof control parameters comprises a first set of zoom control parametersand a first set of focus control parameters for controlling the zoomactuator and the focus actuator, respectively; and wherein the secondset of control parameters comprises a second set of zoom controlparameters and a second set of focus control parameters for controllingthe zoom actuator and the focus actuator, respectively.
 20. An apparatuscomprising: a moveable support; and an optical camera module coupled tothe moveable support, the optical camera module comprising: an opticalassembly including at least one moveable optics; an actuator forpositioning the at least one moveable optics, the actuator including apulley system for moving the at least one moveable optics; a camera forcapturing an image from the optical assembly; and an auxiliary opticsholding multiple optical filters, the auxiliary optics beingpositionable to position one of the multiple optical filters in anoptical path of the at least one moveable optics; and a controllerconfigured to: control the actuator to position the at least onemoveable optics towards a target position according to a first set ofcontrol parameters when the at least one moveable optics is outside athreshold range of the target position; and control the actuator tomaintain the at least one moveable optics at the target position atsteady state according to a second set of control parameters when themoveable optics is within the threshold range of the target position.